The following relates to the retrieval system arts, multimedia document retrieval arts, object annotation arts, and so forth.
Retrieval systems enable selective retrieval of documents (e.g., text documents, images, audio files, video files, multimedia documents such as web pages or word processing documents including embedded images or other embedded non-text content, or so forth) from a database (for example, a dedicated database, or the Internet, or some other collection of documents). Retrieval systems can be useful as stand-alone systems, for example being employed by a user to retrieve documents of interest to the user, or can serve as component of another system. As an illustrative example of the latter, an automatic or semi-automatic image annotation system may employ a retrieval system to identify already-annotated images similar to a new (i.e., query) image, and the annotation system then annotates the new image based on the annotations of the identified similar already-annotated images.
Retrieval systems typically operate on a query basis, in which a query input to the retrieval system by a user or by an application program (e.g., the illustrative annotation system), and the query is compared with documents in the database in order to retrieve relevant documents. A well-known retrieval approach is keyword searching. Here, the documents include text and the query is a keyword or set of keywords, possibly having some logical relationship (e.g., “dog” AND “cat” requiring that a document include both words “dog” and “cat”). The retrieval system retrieves all documents that satisfy the keyword-based query (for example, that contain all the keywords). Keyword-based retrieval is relatively straightforward to implement, but has certain disadvantages, such as: retrieval of an unknown number of documents which can sometimes be prohibitively large; inability to operate on documents that do not contain text (or that do not contain text representing useful informational content); and a strong performance dependence on the user's selection of the keywords.
Other retrieval systems employ a similarity measure, and retrieve the documents found to be “most similar” to the query based on some similarity metric. An advantage of this approach is the number of retrieved documents can be constrained, for example by retrieving a “top-N” most similar documents. Similarity metric-based retrieval systems have other advantages including being less dependent on user selection of keywords, and being usable for documents of various types (not just text).
The query for similarity metric-based searching can be of various types. In an image retrieval system, for example, the user submits an image and the retrieval system compares the image with images in the database to retrieve similar images. In the text domain, the query can be a few words, or a sentence, or even a large document, and the similarity metric compares the text of the query with text of the documents in the database. Such similarity metric-based queries are sometimes called direct relevance queries, because they directly compare the query content with content of the database using a single content medium (e.g., comparison of query text with text of the database documents, or conversely comparison of query image with images of the database documents).
An expansion of direct relevance querying is pseudo-relevance querying. Pseudo-relevance generates a new query based on results retrieved in a first retrieval operation that directly uses the initial query, and then performs a second retrieval operation using the generated new query. For example, an initial text query may be used to retrieve a first set of documents from which text for a new, second query are extracted. The second query is then performed to generate the final results that are associated with the initial text query. Pseudo-relevance can be beneficial because the second query may contain “feedback” words related to, but not contained in, the initial text query, thus enriching the retrieval. For example, an initial text query of “dog breeds” may yield many documents including the word “canine” which was not in the initial query. The second query generated from the first-retrieved documents then includes the word “canine” and may thereby retrieve additional documents that use the related “canine” terminology, but not the “dog” terminology. These additional documents could not have been retrieved by direct operation of the initial query “dog breeds”, even though they may be highly relevant to the initial “dog breeds” query.
Retrieval systems are also known which provide multimedia functionality. These systems retrieve documents that include content of more than one medium. A common example of a multimedia document is a document including both text and images, such as a magazine article, an Internet site having both text and images, or so forth. As another example, a multimedia document may be an Internet site having text, images, and video content (that is three media). One useful multimedia retrieval operation is to employ a query including information represented by two or more different media (e.g., both text and images) and to use the information of the different media in the retrieval. Another useful multimedia retrieval operation uses a query whose content is purely one media (e.g., a stand-alone image) to retrieve database documents based on content in a different media (e.g., text content). To accommodate such tasks, cross-media relevance querying can be employed.
Cross-media relevance, also referred to as trans-media relevance, operates similarly to pseudo-relevance, except that the media type is switched between the initial and second (feedback) query operations. In one example, an initial text query retrieves multimedia documents from which images are extracted and form a second, image query. Cross-media relevance provides similar query expanding benefits as does pseudo-relevance feedback; but, the query expanding capabilities of cross-media relevance extend across different media. For example, cross-media relevance can enable retrieval of a stand-alone image using a purely text query, or conversely may enable retrieval of a pure text document having no images using a stand-alone image query.
The following sets forth improved methods and apparatuses.